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The Afterschool Alliance, in partnership with MetLife Foundation, is proud to present the final issue brief in our 
latest series of four issue briefs examining critical issues facing middle school youth and the vital role afterschool 
programs play in addressing these issues. This series explores afterschool and: the Common Core State Standards, 
students with disabilities and other special needs, data utilization to improve programming, and keeping children 
safe and supported. The briefs examine just a few of the ways afterschool programs support middle school youth, 
families and communities. 


"In my years researching the effects of afterschool 
programs on children's social and academic 
outcomes, I have observed the power that high 
quality programs can have on the learning and 
development of young people." 1 In this 
statement, Dr. Deborah Lowe Vandell highlights 
two important and related points about 
afterschool. The first is the positive impact 
afterschool programs can have on the students 
they serve— impact on a student's academic 
performance, as well as socio-emotional 
development. The second is that the quality of an 
afterschool program is key to student outcomes, 
which is corroborated by findings from other 
studies of the afterschool field. 

For instance, a meta-analysis by the Collaborative 
for Academic, Social and Emotional Learning 
(CASEL) reviewed 68 afterschool program studies 
and compared afterschool programs that had an 
intentional program design grounded in evidence- 
based practices to those that did not. 2 The review 
found that students in the programs that 
employed the evidence-based practices saw 
significant gains in academic performance, school 
day attendance and positive behaviors. The same 
improvements were not seen among students in 
the other set of programs. A separate meta- 
analysis looking only at quality afterschool 
programs found that students regularly 
participating in the programs scored higher on 


standardized math tests, were less likely to 
demonstrate aggressive behavior, and used drugs 
and alcohol less compared to their non- 
participating peers. 3 

A program's structure, design and 
implementation all factor into its impact on the 
students it serves— an afterschool program that 
evaluates its programming and services is able to 
gain valuable information and learn how 
successfully it has implemented its activities and 
achieved the outcomes it set out to accomplish. 4 
A clear vision and mission, collecting data on 
outcomes aligned with goals, assessing impact, 
and making adjustments and adding supports as 
necessary to start the cycle over again are all 
components of a continuous improvement loop 
that can help an afterschool program identify how 
to better meet the needs of their students, 
families and staff. 

What Can Afterschool Programs Gain 
Through Data Collection and Evaluation? 

Data— which include everything from survey 
feedback to attendance records to scores on tests 
of skills and knowledge gained— can serve as a 
valuable resource for afterschool programs, as 
well as their students and families. It is an 
indicator, or indicators, that programs can use to 
improve their understanding of how well their 
offerings are being implemented and what their 
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impact is. It allows programs to identify what their 
strengths are and where changes need to be 
made. 

Afterschool programs that collect data then have 
the opportunity to review and analyze their data, 
which in turn can help a program determine if it is 
carrying out activities and services in the way that 
was intended. If the data do not meet the 
program's expectations, the information gathered 
can be used to determine necessary steps to 
address the issue. Alternatively, if it does 
correspond with what the program envisioned, 
the data can help programs see if those activities 
and services are helping them to accomplish their 
overarching goals, and if other adjustments can 
be made to better meet the needs of their staff, 
students and families. 5 For instance, a three-year 
study that looked at the impact of a continuous 
quality improvement system* found that 
afterschool programs implementing the system 
saw program instruction by staff improve, focus 
on the quality of instruction improve and a small 
positive effect on retaining short-term staff. 6 

Data collection also presents the opportunity for 
afterschool programs to develop and strengthen 
relationships with schools and school districts. For 
Kids Only Afterschool, an afterschool program in 
Massachusetts, worked closely to build and 
maintain a partnership with Everett Public 
Schools— communicating regularly with school 
day staff and sharing attendance and academic 
performance results. 7 

Additionally, data collection and analysis can help 
afterschool programs demonstrate to their 
community, potential funders and policy makers 
that they are making a difference in their 
students' lives— whether it is helping them aspire 
to attend college, building their self-confidence 
and leadership skills, or improving health and 


wellness. Spark Chicago, which is a part of the 
Spark Program with branches in six cities across 
the U.S., was launched in 2011 and served 63 
students. In the course of two years, Spark 
Chicago more than doubled their enrollment 
numbers, serving more than 200 7 th and 8 th 
graders during the 2013-2014 school year. The 
program serves a high need group, more than 9 in 
10 students in the program qualify for the Federal 
Free or Reduced Price Lunch Program and more 
than 80 percent of students are African American. 
Spark Chicago employs a variety of evaluative 
methods to ascertain program data and 
qualitative feedback to improve its model. These 
methods include pre- and post-program surveys 
on a variety of aspects, like self-confidence, 
curiosity, and the ability to communicate 
effectively and ask for help; a data-sharing 
partnership with Chicago Public Schools; and 


“This recent knowledge that high- 
quality afterschool programs work 
and make a positive difference is 
indeed a ‘game changer. ’ This 
means that we should spend much 
less time arguing about whether 
quality after school programs work 
and much more time on working to 
ensure that all programs are 
effective and to make high- quality 
programs more accessible and 
scalable. ” 

- Carol McElvain, director, 
Afterschool and Expanded 
Learning, American Institutes 
for Research 


* The Youth Program Quality Intervention is a continuous improvement model that includes "quality assessment, improvement 
planning, coaching by site managers during staff instruction, and staff attendance at targeted trainings for instructional skill 
building." 
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tracking student attendance and mentor 
retention. Spark Chicago's most recent data show 
that 72 percent of their students reported 
learning how to communicate with adults and 81 
percent stated that participating in the program 
made them more interested in learning new 
things. This sentiment was echoed in parent 
surveys, in which 77 percent of parents noted that 
Spark Chicago made their child more excited 
about learning. Spark Chicago also collects 
longitudinal data. It found that 92 percent of 
apprenticeship program alumni through the class 
of 2012 have completed high school. 


"Current, reliable information is the engine 
of afterschool systems. With reliable data, 
afterschool organizers can chart the 
progress of their efforts, recognize 
problems and possible remedies, handle 
matters ranging from program location to 
program improvements and arm 
themselves with the facts to build a 
compelling case about afterschool for 
funders and the public. In other words, 
good data power good decision making." 

- Better Together: Building Local 
Systems to Improve Afterschool, 
The Wallace Foundation 


What Are the Steps Necessary to Evaluate 
a Program? 

However, before afterschool programs dive into 
the evaluation process, it is important to start first 
with a clear understanding of what they want to 
accomplish, why they want to accomplish it, and 
how they will accomplish it. In her article "Using 
Research to Continuously Improve Afterschool 
Programs: Helping Student to Become 21 st 

Century Lifelong Learners," Denise Huang, project 
director and senior researcher at the National 


Center for Research on Evaluation, Standards, and 
Student Testing at the University of California, Los 
Angeles, outlined the quality indicators of 
afterschool programs under three categories: 
program structure, program content and program 
implementation. 8 

An afterschool program's structure includes a 
mission and vision statement that clearly outlines 
the program's goals, as well as a "theory of 
change" that helps drive the program toward the 
desired long-term goals. 9 Programs that have a 
strong understanding of their mission and vision 
for the future are able to develop the activities 
and services that align with their objectives and 
make progress toward achieving their goals. Just 
as important as program content is program 
implementation, which includes employing strong 
staff who have the ability to lead and run program 
activities, as well as develop positive relationships 
with students and families in the program. Also 
integral is having solid leadership in place that can 
provide direction, motivate staff and make sure 
staff are equipped with the tools and resources 
needed to carry out their duties; and forging 
sound relationships with students, families, 
schools and the community. 

Once a program's quality indicators are defined, 
organized and executed, afterschool programs 
can take the next steps and determine what data 
should be collected in order to review how their 
program is running, observe interactions and 
relationships between staff and students, 
evaluate if their program is having the impact 
they envisioned, and determine if the program is 
moving toward the goals of the organization. 
Common elements of assessment tools measuring 
youth development program quality include 
examining the program's organization and 
curriculum structure; accessibility of activities; the 
program environment; and the relationship 
between program participants and program staff, 
including levels of engagement in activities. 
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What To Measure? 


L 


A unique quality and strength of afterschool programs is that they can be flexible. This means that 
they have the ability to tailor their services to the needs of their students and their community. 
Whether that means creating a community garden and teaching kids about healthy eating habits, 
providing opportunities to learn computer programming skills and work with professionals from the 
community, or helping kids learn to communicate effectively and feel more confident in expressing 
their thoughts and ideas. 

The variety of supports and activities afterschool programs provide and the range of purposes after- 
school programs fulfill also leads to various outcomes they can be interested in documenting. After- 
school programs can partner with schools to track outcomes such as school day attendance, class- 
room behavior, or performance in subjects such as reading and math. But, afterschool programs can 
also focus on other performance indicators, such as responsible decision making, effective communi- 
cation skills, or ability to work in a team. The second set of skills is what the Forum for Youth Invest- 
ment refers to as a "comparative advantage" for afterschool programs, that "represent a strategic 
niche," where afterschool programs can help students develop the 21 st century skills that will prepare 

them for college and work. Wilson-Ahlstrom, A., et. al. (2014). From Soft Skills to Hard Data: Measuring Youth Program Out- 
comes. The Forum for Youth Investment. Retrieved from http://forumfyi.org/files/soft skills hard data O.pdf . 


understanding and acceptance of expectations, 
and responses to students' behaviors. 10 

To help determine what to evaluate, the Harvard 
Family Research Project's afterschool program 
evaluation toolkit recommends that afterschool 
programs consider: 11 

• If the information collected aligns with the 
program's goals and content 

• If the measures make sense in the context of 
the program's lifespan 

• If the data are easily available and accessible 

• If the data collected will be of use to the 
program or its stakeholders 

Similarly, the document "From Good to Great: 
Using Data to Asses and Improve Quality" outlines 
nine tips for afterschool programs to follow to 
help progress from determining what high-quality 
program attributes they would like to focus on to 
the ability to evaluate the system overall. 12 


The Communities Organizing Resources to 
Advance Learning (CORAL) Initiative in California 
is an example of an afterschool program that used 
ongoing evaluation and improvement efforts to 
hone their goals and revise their curriculum, 
linking it to the new program focus, in order to 
better serve their students. 13 Staff training and 
professional development; program monitoring, 
including data collection and staff coaching; and 
data analysis made up CORAL's continuous 
improvement cycle. Once changes were 
implemented, the CORAL Initiative saw program 
quality improve, as well as gains in students' 
reading comprehension. 

Using Data to Improve Programming 

Data collection and use and improving program 
quality are so inextricably linked that funders, like 
The Wallace Foundation, see them as key to the 
future of afterschool providers and citywide 

14 

systems. 
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There are numerous examples of afterschool 
programs across the country that are using data 
to evaluate their programming, determining if 
they are successfully implementing their 
activities and if they have achieved the outcomes 
they hoped to accomplish. By collecting data and 
analyzing the results, programs are using their 
findings to fine-tune their offerings, provide staff 
with the support they need, and ensuring that 
the needs of their students are being met. 

Collecting student information at multiple points 
and in various forms, as well as inputting the 
information regularly into an easily accessible 
platform, helps programs be more responsive to 
the needs of their students. 


Data-Sharing Between Schools and 
Afterschool Programs: Rules of the Family 
Educational Rights and Privacy Act 

Schools have rich sources of student data, such as 
attendance and academic performance records, that 
afterschool programs can use to better meet the needs of 
their students and align programming to build off of 
students' school day lessons. However, an obstacle to data- 
sharing between the two groups has been the Family 
Educational Rights and Privacy Act (FERPA), the federal law 
that protects a student's personally identifiable 
information, such as grades, behavior referrals, social 
security number, and any other information unique to the 
student that could lead to the student's identification. 


Developed by researchers at Northwestern 
University, FUSE provides an interest-driven 
learning experience to engage students across 
Chicago, Illinois, in science, engineering, arts and 
design, and mathematics. FUSE uses a leveling up 
model based on game design to create challenge 
sequences for their students in areas including 
robotics, electronics, graphic design, app 
development and fashion design. FUSE students' 
interests drive the development of challenge 
sequences. For example, a challenge sequence 
designing jewelry using a 3-D computer design 
tool and then constructing a prototype using a 3- 
D printer was created when the program 
discovered students were very interested in 3-D 
design and printing. Students can complete 
challenge sequences individually or in groups and 
have access to FUSE facilitators; scientists, 
engineers and university student mentors; peer 
mentors; and video tutorials to provide guidance 
and support. After finishing a level within the 
challenge sequence, students upload digital 
media artifacts to the FUSE website, where 
others in the program provide feedback. Student 
data— collected through FUSE'S Web platform, in 
-person observations, video observations and 
surveys— is gathered on participation rates at 


In January 2012, FERPA regulations were amended, and 
among the amendments were two clarifications that are 
relevant to the afterschool field: 

1. The definition of an "authorized representative" and an 
"education program" were broadened to include 
programs "principally engaged in the provision of 
education" and "any program that is administered by 
an educational agency or institution" and 

2. The amendment stated that an educational authority 
can "enter into agreement for the purposes of research 
studies." 

The Partnership for Children and Youth's issue brief on the 
changes to FERPA and data-sharing practices between 
schools and afterschool programs found that afterschool 
programs are able to partner with schools and access 
student data by "qualifying as an authorized representative 
of a contracted education program" and "conducting 
studies in partnership with schools." The issue brief also 
outlines various ways afterschool programs are able to 
work with schools within FERPA's requirements and 
successfully create data-sharing systems that allow 
programs to tailor their curriculum to the needs of their 
students, supporting students in the areas where they need 
it most. 

Partnership for Children and Youth. (2012). "Data-Sharing: Federal Rules 
and Best Practices to Improve Out-of-School-Time Programs and Student I 
Outcomes." Retrieved from http://www.partnerforchildren.org/storage/ j 
documents/downloads/after school downloads/ost data- ji 
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each site, what challenges students participate in 
and how long each student spends on challenges. 
Data collected help FUSE recognize what 
challenge sequences are most appealing to 
students, what activities sustain student 
engagement and why, new challenge sequences 
that can be created, and how to better support 
the development and continuation of learning 
pathways. 

Partnering with other institutions can help on 
various fronts regarding evaluation. Examples 
include partnering with a school district, which 
can give a program access to student data, and 
partnering with a university, which can help a 
program implement an evaluation. 

Citizen Schools North Carolina serves students 
throughout Charlotte and Durham and is a part of 
Citizen Schools, a national nonprofit organization 
that partners with schools serving predominantly 
low-income students and students that are 
struggling academically. Developed to directly 
align with their program model, the Program 
Scorecard tracks both student and staff 
performance. Regarding student performance, 
the Program Scorecard includes metrics for 
student attendance, English language arts and 
math test scores and grades, 21 st century skills, 
beliefs related to college and career preparation, 
and perceptions on self-efficacy. For staff, the 
Program Scorecard tracks staff instructional 
proficiency and apprenticeship quality. Both 
national and regional site staff have access to real- 
time dashboards for student data and Program 
Scorecard reports are available at the end of each 
semester. Each year, Program Scorecard targets 
are set; progress is measured; and annual results 
are compared to determine areas of 
improvement, areas in need of work and 
effectiveness of interventions. Through a 
partnership with North Carolina State University, 
researchers from the university conducted a quasi 
-experimental, matched comparison design study 


that tracked students participating in Citizen 
Schools from the Charlotte-Mecklenburg School 
District during the 2007-2008, 2008-2009 and 
2009-2010 school years. Evaluations found that 
Citizen School students were absent from school 
less than students not in the program and 8 th 
grade students in the program were more 
proficient in math and reading than their non- 
participating peers. 


"FERPA doesn't forbid the exchange of 
student information where it is in the 
best interest of kids. As long as 
afterschool providers are sensitive to 
the assurances schools require, in terms 
of how data is shared and used, they 
have every reason to expect they'll be 
granted the access they need." 

- Christopher Kingsley, principal 
associate, data initiatives, 
National League of Cities 


Employing evaluation tools that observe and 
measure various outcomes can lead a program to 
uncover a need they did not know existed. 

BUILD— an in-school and afterschool program 
that provides mentorship, academic enrichment, 
and social and emotional support to at-risk and 
underserved youth in Chicago, Illinois, collects a 
variety of data to ensure that the program is 
meeting the needs of their students and staff. 
BUILD's tools and surveys include: 

• Mutual Accountability Plan (MAP)— helps 
students develop short- and long-term goals, 
identify the steps necessary to reach each 
goal, and monitor and assess each benchmark 
that is reached. 
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• Group Activity Report— measures quantitative 
and qualitative data, recording the activities 
and progress toward achieving the program's 
outcomes. 

• Youth Profile— tracks both students' positive 
and negative behaviors, such as improved 
grades, making progress on goals, substance 
use and behavioral problems. 

• Youth Life Plan Self-Assessment and 
Reflection— asks students to evaluate their 
progress in the program. 

• Health Assessment— collects students' height 
and weight, diet, and exercise habits. 

• Youth Surveys— student surveys are given at 
the end of each program session, providing 
students the opportunity to share their 
opinions on activities, changes they would like 
to see and additional features they would like 
the program to incorporate. 


"We have just started [FUSE] but what I will 
say is that I've never seen a teacher and group 
of students so engaged in instruction in a 
therapeutic school setting. ..The biggest 
changes that I have seen so far are: (1) 
Teachers and students interact as partners in 
problem solving, (2) Students interact with 
other students as partners in problem solving, 
and (3) Students are engaged and excited 
about the class. [It] gets no better from my 
perspective. " 

- Administrator, Special Needs School 


The program implements strategic improvements 
based on the data collected and analyzed through 
their various tools and surveys. For example, 
when BUILD found that 10 percent of their 


students identified as LGBTQ. and that 40 percent 
of students did not know if they had health 
insurance, they reallocated resources to develop 
and implement a new program aimed at 
addressing students' mental health, physical 
health and overall wellness. Program evaluations 
found that 93 percent of BUILD students were 
promoted to the next grade and 93 percent of 
BUILD seniors graduated from high school. 

Conclusion 

The afterschool field has made great gains in 
providing supports and an environment that helps 
their students succeed in school and beyond. 
Students are discovering an interest in science, 
gaining confidence in their speaking abilities, 
missing school less, connecting with caring adults 
and more when participating in quality 
afterschool programs. A distinctive benefit of 
afterschool programs is that their focus and scope 
of activities is wide-ranging— they are able to 
meet the unique needs of their community, which 
could be to offer a dance program in an area 
where arts have been scaled back during the 
regular school day or aligning programming with 
the Common Core State Standards to serve as an 
additional source of support for students, 
teachers and schools. The variety of focus areas 
also means that there are a number of 
measures— both inputs and outputs— that can be 
relevant to their programming. Therefore, it is 
critical that programs have a clear and focused set 
of goals, gather and synthesize data that are 
connected to those goals, and implement a 
continuous improvement cycle that uses the data 
in order to build on lessons learned and ensure 
that they are providing their students with the 
supports they need. Fortunately, more and more 
programs are recognizing the critical role of data 
and more resources and tools are becoming 
available to help programs put the data to use in 
order to best meet the needs of their students. 
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